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TITLE OF THE INVENTION 

STILL IMAGE GENERATING APPARATUS AND STILL IMAGE 
GENERATING METHOD 

5 BACKGROUND OF THE INVENTION 

1. Field of the Invention 

[0001] The present invention pertains to a technique of generating still 
image data having a relatively high resolution from multiple image data 
having a relatively low resolution. 

10 2. Description of the Related Art 

[0002] Moving image data that is captured and recorded by a digital 
video camera or the like contains multiple relatively low-resolution image 
< data (such as frame image data and the like). In the conventional art, one 
frame image data is obtained from this moving image data and is used as a 

15 still image. When frame image data is obtained from moving image data, 
still image data having a higher resolution is generated by obtaining and 
combining multiple frame image data and performing image synthesis by 
interpolating the pixel data. The method by which the multiple frame image 
data are combined and synthesized in this fashion can be expected to result 

20 in higher image quality than a method in which one frame image undergoes 
resolution conversion. Here, 'resolution' refers to the density or number of 
pixels constituting one image. 

[0003] As a technology to create the still image data described above, 
Japanese Patent Laid-Open No. Hll- 164264, for example, discloses a 
25 technology by which a high-resolution image is generated by selecting from 
among (n+l) continuous frame images a frame image as a reference frame 
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image, calculating the movement vectors for the other (n) frame images 
(target frame images) relative to this reference frame image, and 
synthesizing the (n+l) frame images based on each of these movement 
vectors. 

5 [0004] However, where a high-resolution image is created through 
synthesis of multiple low -re solution frame image data as described above, 
because the processing time is much longer than the time needed for 
creation of a high-resolution image via interpolation of image data from one 
frame image, demand exists in the art for a shortening of such processing 
10 time. 

In addition, such demand exists not only where still images are 
obtained from moving image data as described above, but also where such 
images are obtained simply from multiple image data. 

15 SUMMARY OF THE INVENTION 

[0005] Therefore, in view of the foregoing, an object of the present 
invention is to provide a technology that offers reduced processing time 
when image synthesis is performed using multiple image data. 
[0006] In order to achieve at least a part of the above object, the still 

20 image generating apparatus of the present invention is a still image 
generating apparatus that generates still image data from multiple image 
data, comprising* 

an image acquisition unit that obtains from among the multiple 
image data multiple first image data that are arranged in a time series; 
25 an image storage unit that stores the multiple first image data 

obtained by the image acquisition unit; 
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a correction amount estimating unit that estimates with regard to 
the multiple first image data stored in the image storage unit the amount of 
correction required to correct for positional deviation among the images that 
are expressed by the multiple first image data; and 
5 an image synthesizer that corrects the positional deviation among 

the images expressed by the multiple first image data based on the 
estimated correction amounts, and synthesizes the corrected multiple first 
image data to generate as the still image data second image data having a 
higher resolution than said first image data. 

10 [0007] When second image data having a higher resolution than the first 
image data is generated as described above, because there is no longer any 
need to obtain once more from the multiple image data multiple first image 
data arranged in a time series, and second image data can be generated 
using the multiple first image data stored in the image storage unit, the 

15 time required for processing can be reduced accordingly. 

[0008] The multiple image data described above may include moving 
image data. In this case, the still image data can be generated from moving 
image data. 

[0009] A construction may be employed wherein the image acquisition 
20 unit obtains the multiple first image data from the multiple image data 
when an instruction for image data acquisition is issued, and the image 
storage unit stores the obtained multiple first image data. 
[0010] For example, where the multiple image data constitute moving 
image data, and the file format of this moving image data is random access 
25 format as described below, the multiple first image data can be obtained 
directly from the moving image data. Therefore, the processing described 
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above can be performed when an instruction for image acquisition is issued. 
[0011] It is also acceptable if the image acquisition unit acquires the first 
image data in sequence from among the multiple image data, the image 
storage unit sequentially updates the stored multiple first image data using 
5 the obtained first image data, and the image storage unit maintains the 
stored multiple first image data when image data acquisition is instructed. 
[0012] For example, where the multiple image data constitute moving 
image data, and the file format of this moving image data is sequential 
access format as described below, it is difficult to obtain the multiple first 

10 image data directly from the moving image data, but if the first image data 
is sequentially obtained from the moving image data and the multiple 
stored first image data are sequentially updated using the obtained first 
image data as described above, the multiple first image data can be easily 
acquired when an image acquisition instruction is issued by maintaining the 

15 stored multiple first image data. 

[0013] The image storage unit may also save the second image data 
generated by the image synthesizer in addition to the multiple first image 
data. 

In this case, the generated second image data can be read out and 
20 used at any time. 

[0014] It is acceptable if, where one of several synthesis methods can be 
selectively chosen by the image synthesizer when the corrected multiple 
first image data are synthesized to generate the second image data, the 
image storage unit stores the second image data synthesized using different 
25 synthesis methods separately according to the synthesis method used. 

In this case, the second image data synthesized using different 
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synthesis methods can be read out and used as necessary. 
[0015] It is furthermore acceptable if, where an instruction to 

re -synthesize the corrected multiple first image data using the same 
synthesis method that was previously employed is issued, the image 
5 synthesizer does not synthesize the corrected multiple first image data but 
rather reads out from the image storage unit the second image data that 
was previously synthesized using the same synthesis method described 
above. 

In this case, because identical synthesis processing is not duplicated, 

10 processing time is reduced accordingly. 

[0016] The image storage unit may also save, in addition to the multiple 
first image data, position information indicating the time location within the 
multiple image data of at least one of the multiple first image data obtained. 
[0017] In this case, because the use of saved position information 

15 enables easy access to the time location within the multiple stored image 
data of at least one of the multiple first image data, the processing time 
required to acquire other image data located close to that position within 
the multiple image data can be reduced. 

[0018] It is furthermore acceptable if the present invention includes a 
20 thumbnail image creating unit that creates thumbnail image data from the 
second image data generated by the image synthesizer and an image display 
unit that displays at least the thumbnail image expressed by this thumbnail 
data, and the image display unit displays the thumbnail data together with 
prescribed information concerning the second image data corresponding to 
25 the thumbnail image. 

[0019] In this case, because the user can observe not only the thumbnail 



5 



PF04J791 



image corresponding to the generated second image data, but also 
information concerning the second image data together with the thumbnail 
image, the contents of the generated second image data can be 
comprehensively understood. 
5 [0020] It is furthermore acceptable if, where the image synthesizer can 
selectively choose the synthesis method from among a number of such 
methods when synthesizing the corrected multiple first image data to 
generate second image data, the prescribed information described above is 
information that indicates the synthesis method employed when the second 
10 image data corresponding to the thumbnail image data was generated. 

[0021] In this case, the user can easily learn which of the several 
synthesis methods was used simply from observing this information 
together with the thumbnail image. 

[0022] The present invention is not limited to an apparatus such as the 
15 still image generating apparatus described above, and may be realized in 
the form of a method such as a still image generating method. The present 
invention may furthermore be realized as a computer program that 
implements such method or apparatus, as a recording medium on which this 
computer program is recorded, as data signals that are expressed in a 
20 carrier wave and incorporate this computer program, or in some other form. 

Moreover, where the present invention is realized via a computer 
program or a recording medium on which such computer program is 
recorded, the program may constitute the entire program that controls the 
operations of the apparatus, or may constitute a program that implements 
25 only the functions of the present invention. 

These and other objects, features, aspects, and advantages of the 
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present invention will become more apparent from the following detailed 
description of the preferred embodiments with the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 
5 [0023] Fig. 1 shows the basic construction of the still image generating 
system 100 constituting one embodiment of the present invention; 

Fig. 2 is a block diagram showing the functions of the CPU 11 and 
the RAM 13 of the still image generating system of the above embodiment; 

Fig. 3 is a flow chart showing the sequence of operations performed 
10 during sequential access mode, which constitutes one of the processes 
executed in this embodiment; 

Fig. 4 is a flow chart showing the sequence of operations performed 
during random access mode, which constitutes one of the processes executed 
in this embodiment; 

15 Fig. 5 is a drawing showing the preview screen 200, which is 

displayed on the CRT 18a in this embodiment; 

Fig. 6 is an explanatory drawing of the buffer 140 in this 
embodiment; 

Fig. 7 is a drawing representing the situation wherein a thumbnail 
20 image 221 is generated when the user presses the frame image acquisition 
button 236 in this embodiment; 

Figs. 8(a) through (c) are explanatory drawings showing a data list 
in this embodiment; 

Fig. 9 is a flow chart showing the still image generation process in 
25 this embodiment; 

Figs. 10(a) through (c) are explanatory drawings regarding the 
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selection of the type of still image generating process in this embodiment; 

Fig. 11 is a drawing representing the situation wherein a processing 
type number is entered in connection with a thumbnail image; 

Fig. 12 is an explanatory drawing showing deviation between the 
frame image for the reference frame and the frame image for the target 
frame; 

Fig. 13 is an explanatory drawing showing correction of the 
deviation between the target frame image and the reference frame image; 

Fig. 14 is an explanatory drawing showing the closest pixel 
determination process of this embodiment; 

Fig. 15 is an explanatory drawing that explains the image 
interpolation using the bilinear method in this embodiment; 

Fig. 16 is a drawing representing the situation wherein a balloon is 
displayed with a thumbnail image; and 

Figs. 17(a) and 17(b) are explanatory drawings of the search process 
carried out using the absolute frame number in this embodiment. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0024] An embodiment of the present invention will be described below 

in accordance with the following sequence- 

(l) Embodiment 

A. Still image generating system construction 

B. Summary of processes 

Bl. Overall sequence of processes 
Bl-1. Sequential access mode 
Bl-2. Random access mode 



8 



i 

PF04J791 

Bl-3. Data list creation 
Bl-4. Still image generation 

C. Still image data generation process 
CI. Frame image data acquisition 

5 C2. Correction amount estimation 

C3. Synthesis 

D. Results 
(2) Variation 
[0025] 

10 (1) Embodiment 

A. Still image generating system construction 

Fig. 1 shows the basic construction of the still image generating 
system 100 constituting one embodiment of the present invention. The 
system 100 is composed of a personal computer 10 (hereinafter termed 'PC 

15 100, a digital video camera 30 that can output moving image data, and other 
components. The PC 10 functions as a still image generating apparatus that 
generates frame image data that expresses still images having a relatively 
higher resolution based on multiple relatively low-resolution frame image 
data contained in the moving image data. 

20 [0026] In this embodiment, an image expressed by frame image data is 
also called a frame image. The term 'frame image' refers to a still image that 
can be displayed using the non-interlace method. In addition, the relatively 
high-resolution still image data generated via synthesis of multiple frame 
images is termed generated still image data, and the image expressed by 

25 this generated still image data is termed a generated still image. 

[0027] The PC 10 includes a CPU 11 that executes calculation processes, 
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a ROM 12, a RAM 13, a DVD-ROM drive 15 (hereinafter termed 'DVD drive 
15'), an 1394 I/O 17a, various interfaces (I/F) 17b through 17e, an HDD 
(hard disk) 14, a CRT 18a, a keyboard 18b and a mouse 18c. 
[0028] Stored on the HDD 14 are the operating system (OS), application 
5 programs (APL, including the Application X described below) that can create 
still image data and the like, and other programs. During program 
execution, the CPU 11 forwards the software to the RAM 13 as necessary, 
and executes the program while accessing the RAM 13 as a temporary work 
area. The HDD 14 includes at least a drive area C (hereinafter ? C drive'), a 
10 folder or file storage area under the C drive, and a file storage area under 
the folder area. 

[0029] The 1394 I/O 17a is an I/O port that complies with the IEEE 1394 
standard, and is used to connect to such devices as a video camera 30 that 
can generate and output moving image data. 

15 [0030] The display 18a capable of displaying frame images is connected 
to the CRT 17b, and the keyboard 18b and mouse 18c are connected to the 
input I/F 17c as input devices to enable operation of the apparatus. 
[0031] A printer 20 is connected to the printer I/F 17e via a parallel I/F 
cable. Naturally, the printer 20 may be connected using a USB cable or the 

20 like. 

[0032] A DVD-ROM 15a on which moving image data is recorded is 
inserted in the DVD-ROM drive 15, such that moving image data may be 
read out therefrom. 

[0033] The buffer 140 includes buffer areas 301 through 304 that can 
25 temporarily store frame image data. The RAM 13 includes a data list 
storage area 115 used for storage of the data list described below. 

10 
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[0034] As shown in Fig. 1, the CPU 11 is connected to each component 
via the system bus 10a through execution of the Application X, and performs 
overall control of the PC 10. Fig. 2 is a block diagram showing the functions 
of the CPU 11 and the RAM 13 in the still image generating system of this 
5 embodiment. When the process to generate a generated still image is 
executed, the CPU 11 functions as a frame image controller 110, a frame 
image acquisition unit 111 and a still image generating unit 112. The frame 
image controller 110 controls the various components and performs overall 
control of the processing to generate a generated still image. For example, 

10 when an instruction to play moving images is input by the user via the 
keyboard 18b or the mouse 18c, the frame image controller 110 reads into 
the RAM 13 moving image data from the DVD-ROM 15a loaded in the DVD 
drive 15 or a digital video tape (not shown) constituting a recording medium 
for the digital video camera 30. The frame image controller 110 sequentially 

15 displays on the CRT 18a via the video driver multiple frame images 
contained in the read-in moving image data. As a result, moving images are 
displayed on the CRT 18a. The frame image controller 110 also controls the 
operations of the frame image acquisition unit 111 and still image 
generation unit 112 to generate still image data from frame image data for 

20 multiple frames. The CPU 11 also controls the printing of generated still 
image data by the printer 20. 

[0035] The BIOS that runs the hardware described above, as well as the 
OS and APLs that reside on top of the BIOS, are executed by the PC 10. 
Various types of drivers, such as the printer driver that controls the printer 
25 I/F 17e, are loaded in the OS and control the hardware. The printer driver 
can perform bidirectional communication to and from the printer 20 via the 
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printer I/F 17e, receive image data from APLs, create a print job, and send 
the resulting print job to the printer 20. 

As described above, the still image generating apparatus is 
implemented by both the hardware and software in combination. 
5 [0036] 

B. Summary of processes 

B-l. Overall sequence of processes 

In this embodiment, the Application X can execute various processes 
such as the still image generation process described below. When the user 

10 boots the Application X, the user interface screen (not shown) that enables 
the user to select whether the format of the moving image file to be played is 
sequential access format or random access format is displayed on the CRT 
18a. The frame image controller 110 performs mode switch control based on 
the moving image file format specified by the user. 

15 [0037] Sequential access format is a format in which multiple recorded 
data are accessed according to a fixed sequence. This format is the format 
used when moving image data recorded on a digital video tape is accessed, 
for example. Where the user- specified moving image file format is 
sequential access format, the frame image controller 110 switches to 

20 sequential access mode and executes the sequential access mode process 
shown in Fig. 3. Fig. 3 is a flow chart showing the sequence of operations of 
the sequential access mode process constituting one process executed in this 
embodiment. When this process is executed, the frame image controller 110 
performs control to enable access to the digital video camera 30, which uses 

25 a digital video tape (not shown) as the recording medium. The sequential 
access mode process is explained in detail below. 

12 
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[0038] Random access format is a format in which any desired data 
record is accessed by specifying the position of that data record. This format 
is the format used when moving image data recorded on a DVD-ROM 15a is 
accessed, for example. Where the user-specified moving image file format is 
5 random access format, the frame image controller 110 switches to random 
access mode and executes the random access mode process shown in Fig. 4. 
Fig. 4 is a flow chart showing the sequence of operations of the random 
access mode process constituting one process of this embodiment. When this 
process is executed, the frame image controller 110 performs control to 

10 enable access to the DVD drive 15 in which the DVD-ROM 15a is loaded. 
The random access mode process is explained in detail below. 
[0039] The Application X in this embodiment can interrupt the 

sequential access mode process during mid-processing and switch to the 
random access mode process. It can also interrupt the random access mode 

15 process even during mid-processing and switch to the sequential access 
mode process. Furthermore, the Application X can be terminated when 
necessary even when the sequential access mode or the random access mode 
is active. In any of these situations, the frame image controller 110 performs 
control based on user instructions to interrupt the current mode, switch 

20 among modes, or end the Application X. 
[0040] 

Bl-1. Sequential access mode 

Before the sequential access mode process shown in Fig. 3 is 
explained, the preview screen 200 displayed on the CRT 18a will be 
25 explained. Fig. 5 is a drawing showing the preview screen 200 displayed on 
the CRT 18a in this embodiment. The preview screen 200 shown in Fig. 5 is 
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divided into three areas* a preview area 210, a thumbnail image display 
area 220 and a user instruction area 230. The preview area 210 is a display 
area that plays moving images or displays a frame image as a still image 
when it is specified from among the moving images. The thumbnail image 
5 display area 220 is an area that displays thumbnail images 221 described 
below and the like. The user instruction area 230 contains seven buttons- a 
play button 231, a stop button 232, a pause button 233, a rewind button 234, 
a fast forward button 235, a frame image acquisition button 236 and a still 
image generation button 237. Pressing the play button 231, stop button 232, 

10 pause button 233, rewind button 234 or fast forward button 235 enables the 
moving images in the preview area 210 to be played, stopped, paused, 
rewound or fast forwarded. For example, if the user presses the play button 
231 by moving and operating the mouse cursor 215 using the mouse 18c or 
the keyboard 18d, the frame image controller 110 reads out moving image 

15 data from the video camera 30 and plays the moving images in the preview 
area 210 by displaying the moving image data as moving images in the 
preview area 210. The frame image acquisition button 236 and the still 
image generation button 237 will be explained in detail below. 
[0041] First, when the sequential access mode process shown in Fig. 3 is 

20 executed, the frame image controller 110 first determines whether or not 
moving images are being played in the preview area 210 (step S105). If 
moving images are being played (YES in step S105), the frame images being 
played are buffered in the sequential buffer 140 (step S110). 'Buffering* here 
means the temporary storage of frame image data. This buffering will be 

25 described below with reference to Fig. 6. Fig. 6 is an explanatory drawing of 
the buffer 14 used for buffering of the frame image data from the moving 
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image data in this embodiment. As shown in Fig. 6, the buffer 140 contains 
four buffer areas 301 through 304, and each buffer area is used for buffering 
of the data for one frame image. The frame image data identical to the 
frame image data for the frame image played in the preview area 110 is 
5 buffered in the buffer area 210 by the frame image controller 110. The frame 
image data buffered in the buffer area 301 prior to this buffering is shifted 
to the buffer area 302 and buffered therein. Similarly, the frame image data 
buffered in the buffer area 302 is shifted to the buffer area 303 and buffered 
therein, and the frame image data buffered in the buffer area 303 is shifted 

10 to the buffer area 304 and buffered therein. The frame image data buffered 
in the buffer area 304 is discarded. In this way, the frame image data is 
buffered in the buffer areas 301 through 304 in time-series order. This 
buffering method is called the FIFO (or tunnel stack) method. The frame 
image data buffered in the buffer area 301 is identical to the frame image 

15 data for the frame image being played in the preview area 210, as described 
above, and constitutes the frame image data that serves as the reference 
when multiple frame image data are combined in the synthesizing process 
to generate generated still image data as described below. Therefore, it is 
hereinafter referred to as 'reference frame image data'. 

20 If moving image data is not being played (NO in step S105), the CPU 

II advances to the operation of step S140. 

[0042] The frame image acquisition unit 111 then determines whether or 
not a frame image acquisition operation has been executed (step S115). 
When the mouse cursor 215 is moved and operated by the user so as to press 
25 the frame image acquisition button 236, the frame image acquisition unit 

III determines that the frame image acquisition operation has been 
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performed (YES in step S115) and incorporates into the work area of the 
RAM 13 the four frame image data buffered in the buffer areas 301 through 
304 of the buffer 140 for temporary storage. Where the frame image 
acquisition unit 111 determines that the frame image acquisition operation 
5 has not been executed (NO in step S115), it advances to the operation of step 
S140. 

[0043] The frame image controller 110 records the four frame image data 
that were temporarily saved in the work area of the RAM 13 in a prescribed 
area of the HDD 14 and attaches file names thereto (step S120). In addition, 

10 among the four frame image data temporarily saved in the RAM 13, the 
frame image controller 110 obtains the absolute frame number for the 
reference frame image data buffered in the buffer area 301 by accessing the 
digital video camera 30 (step S125). For example, header information 
indicating the absolute frame number is attached to each frame image data 

15 belonging to the moving image data stored on the digital video tape, and the 
frame image controller 110 may access the digital video camera 30 and 
obtain the absolute frame number for the buffered frame image data as it 
buffers the frame image data from the moving image data in the buffer area 
301, as described above. The absolute frame number is a sequential number 

20 obtained by counting sequentially from the first frame of the digital video 
tape (not shown) constituting a recording medium for the digital video 
camera 30 in this embodiment. 

[0044] Next, the frame image controller 110 uses the reference frame 
image data from among the four frame image data temporarily saved in the 
25 work area of the RAM 13 to create thumbnail image data in the form of a 
bitmap having an 80 x 60 resolution, and displays a thumbnail image 221 in 
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the thumbnail image display area 220, as shown in Fig. 7 (step S130). Fig. 7 
shows a situation wherein a thumbnail image 221 has been generated 
following the pressing of the frame image acquisition button 236 by the 
user. 

5 [0045] The frame image controller 110 then creates a data list used to 
manage various information pertaining to the obtained four frame image 
data, such as the thumbnail image data created in the operation of step 
S130 (step S135 in Fig. 3). The frame image controller 110 saves the created 
data list in the data list storage area 115. 
10 The data list created in this operation will be explained in detail 

below. 

[0046] When creation of the data list is completed, the frame image 
controller 110 determines whether or not the operation to commence the 
still image generation process has been performed (step S140). When the 

15 user moves and operates the mouse cursor 215 to specify a thumbnail image 
displayed in the thumbnail image display area 220 for which to generate a 
still image and presses the still image generation button 237, the frame 
image controller 110 determines that the operation to commence the still 
image generation process has been performed (YES in step S140) and causes 

20 the still image generation unit 112 to execute the still image generation 
process (step S300). 

This still image generation process will be explained below. 
If the frame image controller 110 determines that the operation to 
commence the still image process was not performed (NO in step S140), it 

25 returns to the operation of step S105 and repeats the processing described 
above. 
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[0047] 

B2-2. Random access mode 

If the random access mode process shown in Fig. 4 is executed, on 
the other hand, first, the frame image controller 110 obtains the original 
5 moving image file name for the moving images being displayed in the 
preview area 210 and saves the image data in the RAM 13 after attaching 
the file name (step S200). Specifically, the frame image controller 110 
accesses the DVD drive 15 and obtains the original moving image file name 
from the inserted DVD ROM 15a. 

10 [0048] Next, the frame image controller 110 determines whether or not 
moving images are playing in the preview area 210 (step S203). If moving 
images are being played (YES in step S203), the frame image acquisition 
unit 111 determines whether or not the frame image acquisition operation 
has been performed (step S205). Specifically, if the user moves and operates 

15 the mouse cursor 215 to press the frame image acquisition button 236, the 
frame image acquisition unit 111 determines that the frame image 
acquisition operation has been performed (YES in step S205). Here, the 
frame image acquisition unit 111 obtains the frame image data identical to 
the frame image data represented by the frame image being displayed in the 

20 preview area 210 and the three time-series frame image data for frame 
images displayed in the preview area 210 immediately before this frame 
image from the DVD ROM 15a inserted in the DVD drive 15, and 
temporarily stores the four frame image data in the work area of the RAM 
13. Because among the temporarily saved frame image data the frame 

25 image data identical to the frame image data for the frame image being 
displayed in the preview area 210 is the frame image data constituting the 
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reference where multiple frame image data are combined in the synthesis 
process for generation of a still image described below, it will hereinafter be 
termed the Reference frame image data'. If moving images are not being 
played (NO in step S203), the frame image controller 110 advances to the 
5 processing of step S230 described below. 

[0049] The frame image controller 110 then stores the four frame image 
data temporarily saved in the work area of the RAM 13 in a prescribed area 
of the HDD 14 and attaches file names thereto (step 210). 
[0050] The frame image controller 110 next accesses the DVD drive 15 

10 and obtains the position information for the reference frame image data 
(step S215). For example, header information indicating the position 
information is attached to each frame data belonging to the moving image 
data recorded on the DVD-ROM 15a, and the frame image controller 110 
obtains frame image data from the moving image data as well as the 

15 position information for the obtained frame image data from this header 
information by accessing the DVD drive 15. This position information may 
constitute either an absolute frame image number located on the DVD-ROM 
15a or a number indicating the ordinal position of the frame image in one 
moving image data located on the DVD-ROM 15a. 

20 [0051] When the position information has been obtained, the reference 
frame image data is used to create data for a thumbnail image in the form of 
a bitmap image having an 80 x 60 resolution, and a thumbnail image 221 is 
displayed in the thumbnail image display area 220 as shown in Fig. 15 (step 
S220). 

25 [0052] When thumbnail image creation has been completed, the frame 
image controller 110 creates a data list in which various information 
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regarding the four obtained frame image data are entered, such as the 
thumbnail image data created in the operation of step S220 (step S225). The 
frame image controller 110 saves the created data list in the data list 
storage area 115. 

5 The data list created via this operation will be explained in detail 

below. 

[0053] When data list creation has been completed, the frame image 
controller 110 determines whether or not the operation to commence the 
still image generation process has been performed (step S230). When the 

10 user moves and operates the mouse cursor 215 to specify a thumbnail image 
within the thumbnail image display area 220 for which to generate a still 
image and presses the still image generation button 237, the frame image 
controller 110 determines that the operation to commence the still image 
generation process has been performed (YES in step S230) and causes the 

15 still image generation unit 112 to execute the still image generation process 
(step S300). 

When the still image generation process (step S300) is completed, 
the frame image controller 110 returns to step S200 and repeats the 
processing described above. If the frame image controller 110 determines 
20 that the operation to commence the still image generation process was not 
performed (NO in step 230), it returns to the operation of step S200 and 
repeats the processing described above. 

The still image generation process (step S300) will be described 

below. 
25 [0054] 

Bl-3. Data list creation 
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Here, creation of the data list created in step 135 in sequential 
access mode described above (Fig. 3) and in S220 in random access mode 
described above (Fig. 4) will be explained with reference to Fig. 8. Figs. 8(a) 
through (c) are drawings to explain the data list. Fig. 8(a) is a data list, Fig. 
5 8(b) is a drawing to explain the content associated with the original image 
file format type number, and Fig. 8(c) is a drawing to explain the content 
associated with the processing type number. In Fig. 8(a), the left half of the 
data list indicates the type of data list item, while the right half describes 
the content associated with that type of data list item. 

10 [0055] For the 'frame image acquisition number 1 , the sequential number 
indicating the number of times the frame image acquisition operation (YES 
in step S115 in the sequential access mode process (Fig. 3) and YES in step 
S205 in the random access mode process (Fig. 4)) was performed is entered. 
In Fig. 8, for example, T is entered, indicating the first frame image 

15 acquisition operation. 

[0056] For the 'original moving image file format type number', if the 
original moving image file format constituting the target of the above frame 
image acquisition operation is random access format, T is entered, while if 
the original moving image file format is sequential access format, '2' is 

20 entered, as shown in Fig. 8(b). In Fig. 8, '2' is entered, indicating sequential 
access format. 

[0057] For the 'original moving image file name', only when the original 
moving image file format is random access format, the file name of the 
original moving image file obtained via the operation of step S200 in the 
25 random access mode process (Fig. 4) is entered together with the storage 
path. In Fig. 8, because the format is sequential access format, nothing is 
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entered, and the status is NULL. 

[0058] For the Original moving image position', where the original 
moving image file format is sequential access format, the absolute frame 
number of the reference frame image obtained in the operation of step S125 
5 in the sequential access mode process (Fig. 3) is entered, and where the 
original moving image file format is random access format, the position 
information for the reference frame image obtained in the operation of step 
S215 in the random access mode process (Fig. 4) is entered. In Fig. 8, '300' is 
entered, for example. 

10 [0059] For the 'thumbnail image', where the original moving image file 
format is sequential access format, the actual data for the thumbnail image 
created in the operation of step S130 in the sequential access mode process 
(Fig. 3) is entered, and where the original moving image file format is 
random access format, the actual data for the thumbnail image obtained in 

15 the operation of step S220 in the random access mode process (Fig. 4) is 
entered. 

[0060] For 'still image 1' through 'still image 4', the storage paths and 
file names associated with the four frame image data stored in the 
prescribed area of the HDD 14 are entered. Attached to each file name is a 

20 sequential number. 

[0061] Specifically, where the original moving image file format is 
sequential access format, among the frame image data stored on the HDD 
14 in the operation of step S120 in the sequential access mode process (Fig. 
3), the storage path and file name associated with the frame image data 

25 that was buffered in the buffer area 301 in the operation of step S110 (i.e., 
the reference frame image data) is entered as the still image 1. Similarly, 
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the storage paths and file names associated with the frame image data 
buffered in the buffer areas 302 through 304 are entered as the still images 
2 through 4. 

[0062] Where the original moving image file format is random access 
5 format, on the other hand, the storage path and file name associated with 
the frame image data indicating the frame image displayed in the preview 
area 210 and stored on the HDD 14 in the operation of step S210 in the 
random access mode process (i.e., the reference frame image data) are 
entered as the still image 1, and the storage paths and file names associated 

10 with the three time-series frame image data displayed in the preview area 
immediately before the reference frame image data was displayed in the 
preview area 210 are entered as the still images 2 through 4. 
[0063] The 'processing type number' will be explained in connection with 
the operation of step S350 of the still image generation process described 

15 below (Fig. 9). 

[0064] The data list items 'result of two-frame synthesis 1 , 'result of 
four-frame synthesis' and 'result of one-frame synthesis' will be explained in 
connection with the operation of step S325 of the still image generation 
process described below (Fig. 9). 

20 [0065] 

Bl-4. Still image generation 

Fig. 9 is a flow chart showing the still image generation process in 
this embodiment. The still image generation process (step S300) will be 
explained below with reference to Fig. 9. 

25 When the user specifies a thumbnail image within the thumbnail 

image display area 220 and presses the still image generation button 237, 
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the frame image controller 110 determines that the operation to commence 
the still image generation process has been performed (YES in step S230 in 
Fig. 4), causes the still image generation processing type window 201 shown 
in Fig. 10(a) to appear, and displays it in the preview screen 200 in an 
5 overlapping fashion. 

[0066] Figs. 10(a) through (c) are explanatory drawings showing the 
selection of the type of still image generation processing in this embodiment. 
Fig. 10(a) shows the still image generation processing window 201, while 
Fig. 10(b) shows a sample data list in which the storage paths and file 

10 names of still image data have been entered. Fig. 10(c) shows a situation 
wherein a generated still image is being displayed in the still image 
generation processing window 201. As shown in Fig. 10(a), the preview area 
210 described above is displayed at the left side of the still image generation 
processing window 201, the generated still image display area 250 in which 

15 the generated still image is displayed following still image generation is 
displayed at the right side of the still image generation processing window 
201, the processing type pull-down list 260 from which the user can select a 
type of processing is displayed below these two preview areas, and the 
processing confirmation button 270 is displayed at the bottom right of the 

20 still image generation processing window 201. 

[0067] The user can select a type of synthesis processing from the 
processing type pull-down list 260 (step S305 in Fig. 9). In this embodiment, 
four time-series frame image data were acquired in step S120 in the 
sequential access mode process (Fig. 3) or in step S210 in the random access 

25 mode process (Fig. 4). The process in which synthesis is performed using all 
four of these frame image data and one high-resolution still image data is 
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generated is termed 'four-frame synthesis', the process in which synthesis is 
performed using two frame image data (including the reference frame image 
data) and one high-resolution still image data is generated is termed 
'two-frame synthesis', and the process in which correction is performed 
5 based only on one frame image data (the reference frame image data) and 
one still image data is generated is termed 'one-frame synthesis'. 

The processing involved in 'four-frame synthesis' will be explained in 
detail below. 

[0068] When the user specifies a type of processing from among the 

10 above types of synthesis processing, the frame image controller 110 reads 
out from the data list storage area 130 the data list in which the 
user- specified thumbnail image is stored, and determines in accordance 
with this data list whether or not the user- specified type of processing has 
already been performed (step S310). Here, if the user-specified type of 

15 processing is 'two-frame synthesis', the determination as to whether or not 
such processing has already been performed is made based on whether or 
not a path and file name exist in the 'two-frame synthesis result' field. 
Similarly, if the user-specified type of processing is 'four-frame synthesis', 
determination as to whether or not such processing has already been 

20 performed is made based on whether or not a path and file name exist in the 
'four-frame synthesis result' field, and if the user-specified type of processing 
is 'one-frame synthesis', determination as to whether or not such processing 
has already been performed is made based on whether or not a path and file 
name exist in the 'one-frame synthesis result' field. Specifically, if a path 

25 and file name so exist, it is determined that the user-specified type of 
processing was already performed (YES in step S310), while if no path or 
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file name exist, it is determined that the user- specified type of processing 
has not yet been performed (NO in step S310). 

[0069] If the user- specified type of processing has not yet been 

performed (NO in step S3 10), the frame image controller 110 executes the 
5 specified type of processing (step S315) and stores the generated still image 
data in a prescribed area of the HDD 14 and assigns a file name thereto 
(step S320). The frame image controller 110 then enters the assigned 
storage path and file name in the corresponding data list field for ? two-frame 
synthesis result 1 , 'four-frame synthesis result* or 'one-frame synthesis result' 

10 in accordance with the type of processing specified by the user (step S325). 
For example, if the user specifies 'four-frame synthesis', the frame image 
controller 110 reads out the appropriate data based on the paths and file 
names entered for still images 1 through 4 in the data list and performs the 
four-frame synthesis process described above using these data. The frame 

15 image controller 110 then stores the generated still image data generated 
from the 'four-frame synthesis' process in a prescribed area on the HDD 14 
together with an assigned file name, and enters the storage path and 
assigned file name for the generated still image data in the 'four-frame 
synthesis result' data list field, as shown in Fig. 10(b). 

20 The frame image controller 110 then displays the generated still 

image generated in the operation of step S3 15 in the generated still image 
display area 250 (step S340), as shown in Fig. 10(c). 

[0070] Where the user- specified type of processing has already been 
performed (YES in step S310), the frame image controller 110 reads out 
25 from the HDD 14 the generated still image data that was previously 
generated via the specified type of processing based on the data list in which 
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the user- specified thumbnail image is stored (step S330). For example, 
where the user-specified processing type is Tour-frame synthesis* and that 
processing has already been performed, a path and file name already exist 
in the Tour-frame synthesis result' field of the data list in which the 
5 user- specified thumbnail image is stored. Therefore, the frame image 
controller 110 reads out the frame image data associated with that path and 
file name from the HDD 14. The frame image controller 110 displays this 
generated still image in the generated still image display area 250 (step 
S340). 

10 [0071] The frame image controller 110 then determines whether or not 
processing has been confirmed by the user (step S345). Specifically, when 
the processing confirmation button 270 is pressed, the frame image 
controller 110 determines that processing was confirmed (YES in step S345) 
and enters into the processing type number field in the data list the number 

15 corresponding to the user-specified processing type (step S305) as shown in 
Fig. 8(c) (step S350). For example, where the user-specified processing type 
is f two-frame synthesis', '2' is entered as the processing type number, where 
the user-specified processing type is 'four-frame synthesis', '4' is entered as 
the processing type number, and where the user-specified processing type is 

20 'one -frame synthesis', '1' is entered as the processing type number. In the 
case of 'no processing', '0' is entered. 

[0072] When processing is confirmed, the still image generation 

processing window 201 is closed and the preview screen 200 is displayed. 
Here, the frame image controller 110 displays the processing type number 
25 entered during the operation of step S350 in the thumbnail image 221 for 
which still image generation processing (step S300) was performed. For 
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example, where the processing number specified in step S305 was 
Tour-frame synthesis', the number '4' representing the processing type 
number for 'four-frame synthesis' is displayed in the thumbnail image 221, 
as shown in Fig. 11. Similarly, where the user-specified processing type was 
5 f two-frame synthesis', the number '2 1 representing the processing type 
number for 'two-frame synthesis' is displayed in the thumbnail image 221, 
and where the user-specified processing type was 'one-frame synthesis', the 
number '1' representing the processing type number for 'one -frame 
synthesis' is displayed in the thumbnail image 221. In this way, the user can 

10 learn the last type of processing performed simply by looking at the 
thumbnail image. Where the processing confirmation button 270 is not 
pressed within a prescribed period of time (NO in step S345), the frame 
image controller 110 closes the still image generation processing window 
201 and ends the still image generation process (step S300). 

15 [0073] 

C. Still image data generation process 

The process for generating one relatively high-resolution still image 
data via the 'four-frame synthesis' processing during the still image 
generation process described above (step S300) will be explained below. 

20 [0074] 

Cl. Frame image data acquisition 

Where 'four-frame synthesis' is to be performed in step S315 of the 
still image generation process described above (Fig. 9), the frame image 
controller 110 performs four-frame synthesis by loading into the RAM 13 

25 from the HDD 14 as the four frame image data the frame image data 
associated with the paths and file names in the 'still image 1' through 'still 
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image 4' fields of the data list. 

[0075] The frame image data constitutes the gradation data for each 
pixel based on a dot-matrix system (hereinafter 'pixel data 1 ). The pixel data 
is either YCbCr data composed of Y (brilliance), Cb (blue chrominance 
5 difference) and Cr (red chrominance difference), or RGB data composed of R 
(red), G (green) and B (blue). 

[0076] When four-frame synthesis is begun, first, the still image 

generation unit 112 estimates, under the control of the frame image 
controller 110, the amount of correction needed to correct the 'deviation 1 

10 between the four frame images described above. The term 'deviation* here is 
caused not by the movement of the exposure subjects themselves, but rather 
by changes in the orientation of the camera such as so-called 'panning', or by 
hand shake. In this embodiment, deviation between frame images shifting 
an equal amount for all pixels is assumed. For purposes of estimating the 

15 correction amount, one of the four frame images is selected as a reference 
frame image, and the other three are deemed target frame images. For each 
target frame image, the correction amount required to correct for deviation 
from the reference frame image is estimated. In this embodiment, the image 
that represents the frame image data among the four frame image data read 

20 out as described above that corresponds to the path and file name in the 
'still image 1' field of the data list is deemed the reference frame image. The 
images that represent the frame image data among the four frame image 
data read out as described above that correspond to the paths and file 
names in the 'still image 2' through 'still image 4' fields of the data list are 

25 deemed the target frame images. 

[0077] The still image generation unit 112 then corrects and synthesizes 
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the four read-out frame image data using the sought correction amounts 
and generates still image data from the multiple frame image data. The 
correction amount estimation process and the synthesis process will be 
explained below with reference to Figs. 12 and 13. 
5 [0076] 

C2. Correction amount estimation 

Fig. 12 is an explanatory drawing showing the deviation between 
the reference frame image and the target frame images. Fig. 13 is an 
explanatory drawing showing the correction of the deviation between the 

10 reference frame image and the target frame images. 

[0079] In the explanation below, the symbols F0, Fl, F2 and F3 are 
assigned to the four read-out frame images, and are respectively referred to 
as frame image F0, frame image Fl, frame image F2 and frame image F3. 
Here, the frame image F0 is also referred to as the reference frame image 

15 and the frame images Fl through F3 are also referred to as the target frame 
images. 

[0080] In Figs. 12 and 13, the target frame image F3 is used as a 
representative of the target frame images Fl through F3, and deviation and 
deviation correction are explained with reference to this target frame image 

20 and the reference frame image F0. 

[0081] Image deviation is expressed as a combination of translational 
(horizontal or vertical) deviation and rotational deviation. In Fig. 12, in 
order to make the deviation between the target frame image F3 and the 
reference frame image F0 easy to understand, the sides of the reference 

25 frame image F0 and the sides of the target frame image F3 are overlapped 
onto one another, a hypothetical cross X0 is placed at the center position of 
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the reference frame image FO and a cross X3 is placed at the equivalent 
location on the target frame image F3 to indicate the deviation between the 
reference frame image FO and the target frame image F3. Furthermore, to 
make this deviation amount easy to understand, the reference frame image 
5 FO and cross XO are shown in boldface, while the target frame image F3 and 
cross X3 are shown using dashed lines. 

[0082] In this embodiment, the translational deviation amount in the 
horizontal direction is expressed as 'urn', the vertical translational deviation 
is expressed as Vm f , the rotational deviation is expressed as 'Sin', and the 

10 deviation amounts for the target frame image Fa (where 'a' is an integer 
from 1 to 3) are expressed as 'urna', Vma' and '8ma\ respectively. For 
example, as shown in Fig. 12, the target frame image F3 has both 
translational and rotational deviation relative to the reference frame image 
FO, and these deviation amounts are expressed as 'um3\ Vm3' and ? 8m3\ 

15 [0083] Here, in order to synthesize the target frame images Fl through 
F3 with the reference frame image F0, the position of each pixel in the 
target frame images Fl through F3 must be corrected so as to eliminate any 
deviation between the target frame images Fl through F3 and the reference 
frame image F0. The translational correction amounts used for this 

20 correction are expressed as 'u' in the horizontal direction and V in the 
vertical direction, and the rotational correction amount is expressed as '8\ If 
the correction amounts for the target frame image Fa (where 'a 1 is an integer 
from 1 to 3) are expressed as 'ua f , Va f and '8a\ these correction amounts 'u\ 
V and , 8' relative to the above deviation amounts 'urn', Vm' and '8m' are 

25 expressed using the functions (u— um), (v=-vm) and (8— 8m). The 
correction amounts ua, va and 8a for the target frame image Fa for the 
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frame 'a' are expressed using the functions of (u=— uma), (v=- vma) and 
(8=-8ma). For example, the correction amounts u3, v3 and 83 for the target 
frame image F3 are expressed using the functions of (u— um3), (v=-vm3) 
and (8=-8m3). 

5 [0084] As shown in Fig. 13, by correcting the target frame image F3 
using the correction amounts u3, v3 and 83, the deviation between the 
target frame image F3 and the reference frame image F0 can be eliminated. 
Here, correction means movement of the position of each pixel of the frame 
image F3 by u3 in the horizontal direction, v3 in the vertical direction and 

10 83 in the rotational direction. When this is done, if the corrected target 
frame image F3 and the reference frame image F0 are displayed together on 
the CRT 18a, it is presumed that the target frame image F3 becomes 
partially aligned with the reference frame image F0, as seen in Fig. 13. In 
order to make the results of correction easy to understand, the hypothetical 

15 crosses X0 and X3 used in Fig. 12 are shown in Fig. 13 as well, and it can be 
seen in Fig. 13 that the two crosses are aligned as a result of correction. 
[0085] 'Partially aligned' as described above means that, as seen in Fig. 
13, for example, the hatched area PI is the image for an area that exists 
only in the target frame image F3, and an image for the corresponding area 

20 does not exist in the reference frame image F0. As a result, even where the 
correction described above has been performed, because an image existing 
only in the reference frame image F0 arises due to deviation, and conversely 
an image existing only in the target frame image F3 arises due to deviation, 
the target frame image F3 does not become completely aligned with the 

25 reference frame image F0, but becomes only partially aligned. 

[0086] Similarly, by performing correction to the target frame images Fl 
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and F2 using the correction amounts ul, vl and 81 and u2, v2 and 82, 
respectively, the positions of each pixel of the target frame images Fl and 
F2 can be changed. 

[0087] The correction amounts ua, va and 8a for each target frame image 
5 Fa (where V is an integer from 1 to 3) are calculated as estimated amounts 
by the frame image controller 110 based on the image data for the reference 
frame image F0 and the image data for the target frame images Fl through 
F3 and using a prescribed calculation formula such as the pattern matching 
method or the gradient method, and are transmitted to a prescribed area of 
10 the RAM 13 as translational correction amount data and rotational 
correction amount data. 
[0088] 

C3. Synthesis 

Following completion of correction amount estimation, synthesis 
15 processing is carried out by the still image generation unit 112. The still 
image generation unit 112 first performs correction to the target frame 
image data based on each parameter of the correction amount calculated 
during the correction amount estimation process (Fig. 13). The still image 
generation unit 112 then performs closest pixel determination. 
20 [0089] Fig. 14 is an explanatory drawing showing closest pixel 

determination. While the reference frame image F0 and the target frame 
images Fl through F3 became partially aligned as a result of target frame 
image correction, in Fig. 14, part of each partially aligned image is 
expanded so as to show the positional relationships between the pixels of 
25 the four frame images. In Fig. 14, the pixels of the enhanced high-resolution 
image (generated still image) G are shown as black circles, the pixels of the 
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reference frame image FO are shown as white diamonds, and the pixels of 
the corrected target frame images Fl through F3 are shown as hatched 
diamonds. In this embodiment, the generated still image G is 
resolution-enhanced such that its pixel density is 1.5 times that of the 
5 reference frame image FO. As shown in Fig. 14, the distance between pixels 
of the generated still image G is 2/3 of the distance between pixels of the 
reference frame image FO. Furthermore, the pixels of the generated still 
image G are positioned so as to overlap the pixels of the reference frame 
image FO at every other pixel. However, the pixels of the generated still 

10 image G need not be positioned so as to overlap the pixels of the reference 
frame image FO. For example, it is acceptable if all of the pixels of the 
generated still image G are positioned at various other positions, such as 
between the pixels of the reference frame image FO. Furthermore, the 
resolution enhancement magnification is not limited to 1.5, and may be any 

15 appropriate magnification. 

[0090] Here, focusing on the pixel G(j) representing the jth pixel in the 
generated still image G, first, the distance L0 between this pixel G(j) 
(termed the f focus pixel 1 below) and the pixel belonging to the reference 
frame image FO that is closest to this focus pixel G(j) is calculated. Here, 

20 because the distance between pixels of the generated still image G is 2/3 of 
the distance between the pixels of the reference frame image F0, the 
position of the focus pixel G(j) can be calculated from the position of the 
reference frame image F0. Therefore, the distance L0 can be calculated from 
the position of the reference frame image F0 and the position of the focus 

25 pixel G(j). 

[0091] Next, the distance LI between the focus pixel G(j) and the closest 
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pixel of the target frame image Fl after correction is calculated. Because the 
position of the focus pixel G(j) can be calculated from the position of the 
reference frame image FO, as described above, and the positions of the pixels 
of the post-correction target frame image Fl are calculated during the 
5 correction amount estimation process described above, the distance LI can 
be calculated. Similarly, the distance L2 between the focus pixel G(j) and the 
closest pixel of the target frame image F2 after correction and the distance 
L3 between the focus pixel G(j) and the closest pixel of the target frame 
image F3 after correction are calculated in the same way. 

10 [0092] Next, the distances L0 through L3 are compared with one 
another and the pixel located the smallest distance from the focus pixel G(j) 
(hereinafter the 'closest pixelO is calculated. Because the pixel located at 
the distance L3 is the closest pixel to the focus pixel G(j) in this embodiment, 
as seen in Fig. 14, the pixel of the post-correction target frame image F3 is 

15 determined to be the closest pixel to the reference pixel G(j). Assuming that 
the pixel closest to the focus pixel G(j) was the ith pixel of the 
post-correction target frame image F3 , the pixel is referred to as closest 
pixel F(3,i). 

The above sequence of operations is carried out for all pixels 'j f 
20 (j=l,2,3...) in the generated still image G, and the closest pixel to each such 
pixel is determined. 

[0093] After performing closest pixel determination, the still image 
generation unit 112 performs pixel interpolation. Fig. 15 is an explanatory 
drawing that explains pixel interpretation using the bilinear method in this 
25 embodiment. Because gradation data does not exist for the above focus pixel 
G(j) prior to pixel interpolation, processing to interpolate this gradation data 



35 



PF04J791 



from the gradation data for other pixels is carried out. 

[0094] The gradation data used during the interpolation process is 
composed of the gradation data for the three pixels of the post-correction 
target frame image F3 that surround the focus pixel G(j) together with the 
5 closest pixel (3,i) as well as the gradation data for the closest pixel F(3,i). In 
this embodiment, the gradation data for the focus pixel G(j) is sought based 
on the bilinear method using the gradation data for the pixel F(3,i) closest to 
the focus pixel G(j) and the gradation data for the pixels F(3J), F(3,k) and 
F(3,l) that surround the focus pixel G(j), as shown in Fig. 15. 

10 [0095] While a number of interpolation methods can be used other than 
the bilinear method, such as the bicubic method or the nearest neighbor 
method, an interpolation method that emphasizes the gradation data for the 
pixels closer to the focus pixel G(j) is preferred. Furthermore, the gradation 
data used for this interpolation method should include the data for the 

15 pixels that surround the focus pixel G(j) together with the closest pixel, as 
described above. In this way, by emphasizing the gradation data for the 
pixels closest to the focus pixel and carrying out interpolation using 
gradation data for the pixels close to the closest pixel, gradation data having 
a color value close to the actual color can be established. 

20 [0096] In this way, the still image generation unit 112 performs 

Tour-frame synthesis' during still image generation processing (step S300 in 
Fig. 9), and generates one still image data from the four frame image data 
read out as described above. 

[0097] Where still image data is generated during the still image 
25 generation process described above (step S300 in Fig. 9) via 'two-frame 
synthesis', the frame image controller 110 reads out into the RAM 13 from 
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the HDD 14 the two frame image data corresponding to the paths and file 
names in the 'still image 1' and 'still image 2 f fields in the data list 
(including the reference frame image data), conducts correction amount 
estimation processing and synthesis processing as described above, and 
5 generates one high-resolution still image data. 

[0098] Where still image data is generated during the still image 
generation process described above (step S300 in Fig. 9) via 'one-frame 
synthesis', the frame image controller 110 reads out into the RAM 13 from 
the HDD 14 the reference frame image data corresponding to the path and 
10 file name in the 'still image 1' field in the data list, and generates one 
high-resolution still image data using a pixel interpolation method such as 
the bilinear method, the bicubic method or the nearest neighbor method. 
[0099] 
D. Results 

15 In this embodiment, as described above, four frame image data are 

acquired from the moving image data output by the digital video camera 30 
or the DVD ROM drive 15 and are stored on the HDD 14. As a result, where 
synthesis processing is carried out using multiple frame image data, 
because these multiple frame image data need not be acquired once again 

20 from moving image data output by the digital video camera 30 or the 
DVD-ROM drive 15, and still image data can be generated using the stored 
multiple frame image data, the processing time required to perform image 
synthesis is reduced accordingly. 

[0100] In order to acquire four time-series moving image data from the 
25 moving image data output by the digital video camera 30 in sequential 
access format, the frame image acquisition unit 111 could repeat four times 
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the operation of playing the moving image data and acquiring one frame 
image data each time. However, in this embodiment, in the sequential 
access mode process (Fig. 3), when frame image data is buffered in a time 
series in the buffer areas 301 through 304 of the buffer 140 from the moving 
5 image data playing in the preview area 210 and the user presses the frame 
image acquisition button 236, the buffered frame image data is acquired. As 
a result, because the frame image acquisition unit 111 can acquire four 
time-series frame images without having to repeat the operation of playing 
the moving image data and acquiring one frame image data four times in 
10 succession, the processing time required for generation of still image data 
can be reduced. 

[0101] In this embodiment, when still image data is generated using a 
user- specified type of processing as described above (step S315 in Fig. 9), the 
frame image controller 110 assigns a file name to this data and stores it on 

15 the HDD 14 and enters the file name in the data list. Where the same type 
of processing is to be executed using the same frame images, the still image 
data stored on the HDD 14 is read out in accordance with the data list and 
is displayed in the generated still image display area 250. As a result, 
because the frame image controller 110 need not perform the same 

20 processing once more, the processing time can be reduced. 

[0102] The frame image controller 110 displays the processing type 
number in the thumbnail image as described above. As a result, the user 
can learn the type of synthesis processing last performed simply by looking 
at the thumbnail image. The present invention is not limited to this 

25 implementation, and it is acceptable if a prescribed symbol is displayed in 
the thumbnail image to indicate the type of synthesis processing last 
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performed. For example, a construction may be adopted wherein a circle is 
displayed if the last performed synthesis method was 'one-frame synthesis*, 
a triangle is displayed if the last performed synthesis method was 
'two-frame synthesis* and a square is displayed if the last performed 
5 synthesis method was 'four-frame synthesis'. Alternatively, prescribed 
information could be displayed in the thumbnail image. Moreover, a balloon 
may be used as the method for displaying this prescribed information. For 
example, when the mouse cursor 215 is placed over the thumbnail image 
221 created in the thumbnail image display area 220, a balloon containing 

10 prescribed information can be displayed, as shown in Fig. 16. The prescribed 
information displayed in the balloon 229 in this example includes the 
original moving image position and the types of [synthesis] processing 
performed. In this way, the user can see prescribed information such as the 
original moving image position or the types of processing previously 

15 performed simply by moving the mouse cursor 215 over the thumbnail 
image. 

[0103] Because the frame image controller 110 stores the absolute frame 
number for the reference frame image obtained in step S125 of the 
sequential access mode process (Fig. 3), the search operation described 

20 below can be performed. 

Figs. 17(a) and 17(b) are explanatory drawings regarding a search 
operation using an absolute frame number in this embodiment. As shown in 
Fig. 17(a), thumbnail images 221 and 222 are being displayed in the 
thumbnail image display area 220 of the preview screen 200, and a frame 

25 image that differs from the images represented by the thumbnail images 
221 and 222 is being displayed in the preview area 210. 
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[0104] When the user then specifies a thumbnail image for which a 
search is to be performed, the data list in which that thumbnail image is 
stored is read out and the absolute frame number for the 'original moving 
image position* in the data list is obtained. The frame image controller 110 
5 then accesses the digital video camera 30 and rewinds or fast forwards the 
digital video tape (not shown) until the frame image located at the position 
corresponding to the obtained absolute frame number is reached. As a result, 
the frame image located at the position corresponding to the specified 
absolute frame number can be displayed in the preview area 210, as shown 
10 in Fig. 17(b). In addition, because the moving images can be played, fast 
forwarded or rewound from this position, frame image data located near this 
position can be acquired once more. 

[0105] Because the frame image controller 110 stores the position 
information for the reference frame image obtained in step S215 in the 

15 random access mode process (Fig. 4), searching can be carried out. 
Specifically, when the user specifies a thumbnail image for which a search is 
to be performed, the frame image controller 110 reads out from the storage 
area 130 the data list in which the thumbnail image is stored. The frame 
image controller 110 then obtains the position information from the 'original 

20 moving image position' field of that data list. In addition, the frame image 
controller 110 accesses the DVD-ROM drive 15 and acquires the frame 
image located at the position corresponding to the obtained position 
information. As a result, the frame image located at the position 
corresponding to the position information can be displayed in the preview 

25 area 210. Furthermore, because the moving images can be played, fast 
forwarded or rewound from this position, the frame image data located near 
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this position can be acquired once more. 

[0106] Because the absolute frame number of the reference frame image 
obtained in step S125 of the sequential access mode process (Fig. 3) or the 
position information for the reference frame image obtained in step S215 of 
5 the random access mode process (Fig. 4) are stored, where multiple 
thumbnail images are being displayed in the thumbnail image display area 
220, the frame image controller 110 can sort these multiple thumbnail 
images in a time series based on the absolute frame number or position 
information. 

10 Normally, the thumbnail images displayed in the thumbnail image 

display area 220 are displayed in the order of their creation, and the user 
cannot readily determine the time-series relationships within the moving 
image data of the images corresponding to each thumbnail. Therefore, when 
the user issues an instruction to perform sorting of the thumbnail images, 

15 the frame image controller 110 reads out from the data list storage area 130 
the data lists in which the thumbnail images displayed in the thumbnail 
display area are stored and performs sorting according to the values in the 
'original moving image position* fields of these data lists. This enables the 
user to display the thumbnail images in the thumbnail image display area 

20 220 in time-series order. 
[0107] 

(2) Variation 

The present invention is not limited to this implementation, and 
various other constructions within the essential scope of the invention may 
25 be employed. 

[0108] In the above embodiment, the method for buffering data in the 
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buffer 140 was the FIFO method, but the present invention is not limited to 
this method. For example, the buffer 140 may be a ring buffer. In this case, 
the frame image being played in the preview area 210 may be buffered by 
sequentially overwriting the buffer area of the buffer 140 in which the oldest 
5 frame image is buffered. In addition, the buffer 140 in the above 
embodiment may be disposed in a prescribed area of the RAM 13. 
[0109] In the above embodiment, moving image data was read out from 
the digital video camera 30 or DVD-ROM drive 15 and multiple frame image 
data belonging to this moving image data were acquired and stored in the 

10 buffer 140, the RAM 13 or the HDD 14, but the present invention is not 
limited to this implementation. It is also acceptable if the moving image 
data is read out from a recording medium connected to the PC 10, such as a 
magneto-optical disk, CD-R/RW disk, DVD or magnetic tape, and multiple 
frame image data contained in this moving image data are acquired and 

15 stored in the buffer 140, RAM 13, HDD 14 or the like. 

[0110] In the still image generating system of the above embodiment, 
the frame image data to be acquired is two or four frames of frame image 
data that are continuous in a time series from the time at which the 
instruction for acquisition is issued, but the present invention is not limited 

20 to this implementation. The frame image data to be acquired may be frame 
image data for three frames or for five or more frames. In this case, it is 
acceptable if the processing to generate relatively high-resolution still image 
data is performed using some or all of the acquired frame image data. 
[0111] In the above embodiment, a situation was described wherein one 

25 relatively high-resolution still image data was generated by acquiring 
multiple frame image data that are continuous in a time series from among 
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the moving image data, and synthesizing these frame image data, but the 
present invention is not limited to this implementation. It is also acceptable 
if one relatively high-resolution still image data is generated by acquiring 
multiple frame image data that are arranged but non-continuous in a time 
5 series from among the moving image data and synthesizing these frame 
image data. It is also acceptable to generate one relatively high-resolution 
still image data simply by acquiring multiple frame image data that are 
arranged but non-continuous in a time series from among multiple frame 
image data that are continuous in a time series, and synthesizing these 
10 frame image data. Such multiple image data that are continuous in a time 
series may comprise multiple image data captured by a digital camera via 
rapid shooting, for example. 

[0112] In the above embodiment, a personal computer was used as the 
still image generating apparatus, but the present invention is not limited to 

15 this implementation. The still image generating apparatus described above 
may be mounted in a video camera, digital camera, printer, DVD player, 
video tape player, hard disk player, camera-equipped cell phone or the like. 
In particular, where a video camera is used as the still image generating 
apparatus of the present invention, one high-resolution still image data can 

20 be generated from multiple frame image data included in the moving image 
data for the moving images captured by the video camera at the same time 
as capture of moving images occurs. Furthermore, where a digital camera is 
used as the still image generating apparatus of the present invention, one 
high-resolution still image data can be generated from multiple captured 

25 image data while shooting of the photo object occurs or as the user confirms 
the result of image capture of the photo object. 
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[0113] In the above embodiment, frame image data was used as an 
example of relatively low-resolution image data, but the present invention is 
not limited to this implementation. For example, the processing described 
above may be carried out to field image data instead of to frame image data. 
5 Field images expressed by field image data are even-numbered and 
odd-numbered still images in the interlace method that comprise images 
equivalent to frame images in the non-interlace method. 
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